-
Notifications
You must be signed in to change notification settings - Fork 0
TBE-198: limit number of login sessions with MeF #76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
TBE-198: limit number of login sessions with MeF #76
Conversation
…of threads to control concurrency of webhook callback jobs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am curious how you landed on 15 rather than 10 or 20?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The existing value was 5, and I figured that (if I want all connection attempts to succeed immediately) I'm adding 10 new possible threads that might need a DB connection (5 single-threaded workers for the MeF jobs, 1 5-threaded worker for the callbacks).
I'm not actually sure of myself here (maybe the webhook callback worker actually only needs one db connection, since the job doesn't use the db and therefore the only need is for the worker to claim & execute jobs?) but that was my reasoning, faulty or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you running the queue embedded in the puma server or are you running the jobs queue stand alone? If running stand alone, I don't think you should need to change this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm also unsure though about how many threads are reasonable for postgres. I've never really deviated from the default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re puma server: i would imagine stand alone since there would be additional configuration for puma
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
queue is puma plugin in development, standalone elsewhere - see
Line 37 in c3774ac
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development? |
The actual error I get when running bin/rails s
in development is:
Solid Queue is configured to use 7 threads but the database connection pool is 5. Increase it in `config/database.yml`
I assume the 7-thread requirement is coming from the 5-threaded webhook callback worker, per this note in the solid_queue docs:
It is recommended to set this value less than or equal to the queue database's connection pool size minus 2, as each worker thread uses one connection, and two additional connections are reserved for polling and heartbeat.
I'm going to resolve this by reducing the number of threads that the webhook callback worker uses to 3, rather than increasing the db connection pool, since I'm also uncertain about the impact of the latter option and the former should be relatively low-impact (since our webhook callback jobs shouldn't take long to complete, since I'm also adding a 5-second timeout to all webhook callback requests, down from the default of 60(!))
what is the way to go about testing this? |
… to remain at default, and add a timeout of 5 seconds to webhook callback requests
I've tested it manually, and just did so again on video! Let me know if you can get to this @anisharamnani @powersurge360 https://codeforamerica.zoom.us/clips/share/7aa-RyQ8Szy6-WgN3N-9Tg |
It turns out reality is not so simple - I just updated the MeF SDK so that we can make actual requests to ATS again and ran this same test, and got several |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks mike for your thorough testing video! greatly appreciated :)
use number of workers to control concurrency of mef jobs, and number of threads to control concurrency of webhook callback jobs